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Chapter  1 


INTRODUCTION 

1.1     Background 

One  of  the  major  problems  facing  the  construction  industry  in  the  coming 
years  is  its  continual  decline  in  productivity.  A  recurring  problem  since  the  mid 
1960s,  the  extent  of  the  decline  continues  to  elude  researchers  and  industry  officials. 
The  inability  to  determine  the  extent  of  the  decline  has  been  attributed  by  the  Buisness 
Roundtable  to  the  diverse  and  fragmented  nature  of  the  industry,  as  well  as  the 
absence  of  a  standard  method  of  measurement  and  an  industry  wide  data  base.^  In 
examining  the  productivity  issue  it  is  obvious  that  all  aspects  of  design,  construction 
and  project  management  must  be  reviewed. 

A  general  consensus  exists  that  as  a  project  progresses  from  feasibility  studies 
and  preliminary  design,  to  detailed  design,  procurement,  and  construction,  the 
amount  of  control  or  influence  that  can  be  exerted  on  the  project  decreases.  Decisions 
made  early  in  the  design  phase  have  the  largest  impact  on  not  only  total  construction 
cost,  but  often  on  the  constructability,  maintainability,  and  operabihty  of  the  facility. 
Typically,  the  cost  of  the  design  phase  represents  less  than  10%  of  the  total 
construction  costs.  Due  to  their  large  impact  and  relatively  small  cost,  it  is  essential 
that  the  products  of  the  design  phase  be  examined  when  productivity  improvements 
are  desired. 


The  Construction  Industry  Institute  has  identified  a  matrix  evaluation 
technique  known  as  the  Objectives  Matrix  as  a  tool  for  evaluating  the  effectiveness  of 
design.  The  matrix  format  allows  the  evaluation  of  many  diverse  factors  that  are 
normally  difficult  to  quantify  and  produces  an  index  which  is  used  to  track  and 
evaluate  performance. 

1.2  Purpose 

The  purpose  of  this  study  is: 

1.  to  explore  the  applications  of  the  design  evaluation  matrix  in 
commercial  building  and  heavy  civil  construction. 

2.  to  review  guidelines  for  implementing  the  design  evaluation 
matrix. 

3.  to  make  recommendations  for  additional  criteria  and  sub-criteria. 

1.3  Scope 

This  study  is  a  continuation  of  a  Construction  Industry  Institute  study  on  the 
evaluation  of  design  effectiveness.  Phase  I  of  the  research  introduced  the  Design 
Evaluation  Matrix,  a  variation  of  the  Objectives  Matrix,  as  a  technique  for 
determining  design  effectiveness. ^  In  phase  II,  the  components  of  the  Design 
Evaluation  Matrix  were  refined  through  a  study  of  the  piping  phase  of  large  industrial 
projects. 3  The  evaluation  criteria  and  sub-criteria  used  in  this  study  are  based  on 
those  established  in  phase  I  of  the  research.  These  criteria  are:  accuracy  of  the  design 
documents,  usability  of  the  design  documents,  cost  of  the  design,  constructability, 
performance  against  design  schedule,  economy  of  the  design,  and  ease  of  occupancy  / 
start-up. 


1.4     Organization 

This  report  briefly  reviews  the  Design  Evaluation  Matrix  as  an  evaluation 
technique  for  design  effectiveness.  A  review  of  the  matrix  theory  and  matrix 
construction  is  presented  in  Chapter  2.  In  Chapter  3  the  evaluation  criteria  and  sub- 
criteria  are  discussed.  The  feasibility  of  applying  the  method  to  commercial  building 
and  heavy  civil  construction  is  explored  in  Chapter  4.  A  Design  Evaluation  Matrix 
example  for  a  commercial  building  is  presented  in  Chapter  5  using  the  weights  and 
performance  levels  collected  from  the  various  projects  volunteered  for  the  study. 
Chapter  6  presents  the  conclusions  of  the  study  and  recommendations  for  the  report. 
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Chapter  2 


THE  DESIGN  EVALUATION  MATRIX 

In  an  attempt  to  improve  productivity  throughout  the  construction  industry 
many  Industrial  Engineering  techniques  used  in  manufacturing  industries  such  as  time 
and  motion  studies  and  time-lapse  photography,  have  been  adapted  for  use  by  the 
construction  industry.  Another  method  receiving  increasing  attention  is  matrix 
measurement.  Matrix  measurement  does  not  directiy  measure  productivity.  Instead, 
the  factors  that  affect  production  time,  quality,  and  cost  are  measured.  One  form  of 
matrix  measurement  known  as  the  Objectives  Matrix  provides  the  means  to  list, 
categorize,  and  weight  the  key  indicators  of  performance  that  relate  to  the  objectives 
of  a  project.  A  variation  of  this  method,  called  the  Design  Evaluation  Matrix,  is  being 
researched  by  the  Construction  Industry  Institute  for  evaluating  design  effectiveness. 

2.1     Introduction 

There  are  four  main  components  of  an  evaluation  matrix,  the  criteria  (sub- 
criteria),  the  weights,  the  performance  scale,  and  the  performance  index. 

The  criteria  (sub-criteria)  define  what  is  to  be  measured.  The  weights 
determine  the  relative  importance  of  the  criteria  (sub-criteria)  to  the  operation  of 
interest.  The  performance  scale  compares  the  measured  value  of  a  specific  criterion 
on  a  project  to  past  performance  or  to  future  goal  for  that  criterion.  From  these  three 


'      1  ' 


components  the  performance  index  is  calculated  and  used  to  evaluate  and  track 
perfonnance. 

2.2    Construction  of  The  Matrix 

This  section  is  provided  a  guide  on  how  to  construct  an  evaluation  matrix,  and 
to  illustrate  how  the  components  of  the  matrix  interact.  Figure  2. 1  should  be  referred 
to  throughout  the  explanation  of  matrix  construction. 

The  first  step  in  constructing  a  matrix  is  the  selection  of  the  evaluation  criteria. 
In  developing  the  Objectives  Matrix,  Riggs'  proposed  three  guidelines  for  selecting 
evaluation  criteria.^  First,  the  criteria  should  pertain  only  to  the  activities  that 
contribute  directly  to  the  performance  goals.  Second,  all  goals  should  be  reasonable, 
and  their  attainment  dependent  only  upon  the  actions  of  the  group  being  measured. 
Finally,  the  criteria  should  be  representative  of  all  of  the  work  responsibilities  of  the 
group. 

Once  the  criteria  (sub-criteria)  have  been  selected  weights  are  assigned  by 
management  in  accordance  with  its  perception  of  how  each  criterion  impacts 
performance.  In  the  weighting  process,  100  points  are  distributed  among  the  criteria. 

Notice  that  a  performance  scale  of  0  to  10  has  been  drawn  on  the  right  side  of 
the  matrix.  A  score  of  10  represents  a  future  goal  that  is  attainable  in  the  foreseeable 
future  with  current  resources.  A  score  of  3  is  designated  as  average  (rather  than  a 
score  of  5)  to  allow  more  room  for  improvement.  A  score  of  0  represents  the 
minimum  acceptable  level  of  performance  based  on  recent  experience.  These  three 
levels  represent  the  "benchmark"  levels  for  the  matrix. 

After  the  values  for  the  benchmark  scores  are  established,  values  are 
determined  for  the  levels  between  benchmarks.  These  values  may  be  established  by 
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FIGURE   2.1 


one  of  three  different  techniques,  numerical  increment,  percentage  increment,  or  a 
subjective  ratio.  The  technique  used  will  depend  on  the  nature  of  the  criterion  and  the 
workers  being  evaluated. 

In  the  numerical  increment  technique  each  level  on  the  performance  scale 
differs  by  the  same  numerical  value.  For  example,  the  increment  between  scores  one 
and  two  is  equal  to  the  increment  between  scores  eight  and  nine.  These  numerically 
linear  increments  result  in  an  increasingly  larger  percentage  of  improvement  with  each 
upward  movement.  A  formula  to  determine  the  numerical  increment  for  values 
between  the  benchmark  levels  is  presented  in  Appendix  1 . 

The  percentage  increment  technique  utilizes  a  non-hnear  scale.  This  technique 
produces  increments  in  which  the  percentage  difference  between  any  two  consecutive 
levels  is  the  same.  A  formula  to  determine  the  percentage  increment  scale  is  also 
contained  in    Appendix  1 . 

After  all  values  have  been  entered  in  the  appropriate  columns  and  rows  of  the 
matrix,  the  project  scores  assigned  to  each  criterion  are  entered  in  the  row  under  the 
criteria  labeled  "performance". 

In  each  column  the  score  closest  to  the  project  score  that  does  not  exceed  it  is 
circled.  If  a  measured  score  falls  between  matrix  scores  it  is  assigned  the  lower  of  the 
two  scores.  A  score  is  not  assigned  until  it  has  been  attained. 

The  circled  scores  are  recorded  in  the  box  below  each  criterion  labeled 
"score".  This  score  is  multiplied  by  the  criterion  weight,  and  the  product  is  recorded 
in  the  row  labeled  "value".  After  all  values  have  been  determined  they  are  added 
together  to  obtain  the  "performance  index". 


2.3     Sub-Matricies 

The  use  of  matrix  measurement  allows  each  criterion  on  the  master  matrix  to 
be  individually  measured  by  the  use  of  sub-matricies.  A  sub-matrix  is  developed  for 
each  criterion  on  the  master  matrix  in  the  same  manner  that  the  master  matrix  was 
developed.  Sub-criteria  are  much  more  specific  than  the  performance  criteria  on  the 
master  matrix.  Once  completed,  the  performance  index  of  a  sub-matrix  becomes  the 
measured  value  for  the  appropriate  criterion  on  the  master  matrix.  This  enables  the 
user  to  track  the  performance  of  a  specific  criterion  by  tracking  the  performance  index 
of  the  corresponding  sub-matrix. 

An  example  of  a  sub-matrix  and  its  relationship  to  a  criterion  on  the  design 
evaluation  matrix  is  shown  in  Figure  2.2. 
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The  index  for  the  submatrix  becomes  the 
project  measured  value  for  the  performance 
criterion  on  the  master  matrix. 


Relationship  Between  Design  Evaluation  Matrix  and  Sub-Matrix 


FIGURE   2.2 
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Chapter  3 


DESIGN  EVALUATION  CRITERIA 

In  order  for  the  design  evaluation  matrix  to  be  utilized  effectively,  evaluation 
criteria  must  be  identified  that  relates  to  the  design  function  and  to  potential 
construction  and  maintenance  problems  that  may  result  from  an  improper  design.  In 
its  effort  to  develop  the  design  evaluation  matrix,  the  Construction  Industry  Institute 
has  identified  fourteen  evaluation  criteria  that  are  generally  applicable  to  all  types  of 
construction  projects.  These  evaluation  criteria  were  provided  to  owners, 
constructors  and  designers  as  a  starting  point  for  this  investigation. 

3.1     Criteria  Selection 

Design  is  a  complex  function  and  will  require  a  variety  of  performance  criteria 
to  effectively  evaluate  it.  Many  variables  will  influence  the  actual  selection  of  design 
criteria  and  sub-criteria.  Since  each  project  is  unique  to  a  degree,  the  selection  of 
criteria  needs  to  be  tailored  to  each  project.  Variables  influencing  criteria  selection 
include  the  type  of  project,  type  of  design  contract,  type  of  construction  contract, 
scheduled  duration  of  the  design  and  construction  periods,  and  the  project's  budget. 
Additionally,  the  differing  perspectives  of  the  designer,  constructor,  and  owner  need 
to  be  considered  in  selecting  evaluation  criteria  and  sub-criteria.  Some  of  the 
designer's  principal  concerns  are  a  design  that  meets  all  owner  requirements  and 
regulatory  codes,  is  aesthetically  pleasing,  and  enhances  its  professional  reputation. 
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The  constructor  is  concerned  with  the  constructability  of  the  design,  the  accuracy  of 
the  design  products,  and  its  ability  to  meet  the  owner's  schedule,  as  well  as  the  ability 
to  generate  a  profit.  The  owner's  concerns  include  the  items  mentioned  above  along 
with  the  long  range  aspects  of  the  design,  such  as  life-cycle  costs,  maintainability, 
and  operability. 

3.2     Design  Evaluation  Criteria 

During  the  initial  phase  of  its  study,  the  Construction  Industry  Institute 
identified  fourteen  design  criteria.  ^  With  only  minor  modifications  these  criteria  can 
be  utilized  on  most  construction  projects.    These  criteria  are  presented  in  Figure  3.1. 

Design  Evaluation  Criteria 

Accuracy  of  the  Design  Documents 

Completeness  of  the  Design  Documents 

Clarity  of  the  Design  Documents 

Usability  of  the  Design  Documents 

Economy  of  the  Design 

Timeliness  of  the  Design 

Start-up  Costs  and  Time  Required  •    >      < 

Cost  of  the  Design  Effort 

Constructability  of  the  Design 

Operability  of  the  Design  Facility 

Maintainability  of  the  Design  Facility 

Safety  of  the  Design 

Plant  Operating  Efficiency 

Plant  Performance 

Figure  3.1 

The  process  of  design  evaluation  should  be  an  ongoing  endeavor  and  can  be 
divided  into  three  phases.  In  phase  one,  efforts  are  made  to  track  the  effectiveness  of 
the  design  during  the  design  process.   The  injection  of  contractor  experience  and 
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knowledge  can  play  a  significant  role  in  improving  the  constnictability  and  cost 
effectiveness  of  a  design. 

In  the  second  phase,  construction,  the  design  is  evaluated  for  constnictability, 
accuracy,  clarity,  and  information  content.  Studies  have  shown  a  direct  relationship 
between  clarity,  information  content,  and  overall  quality  of  a  design,  and  the 
productivity  of  construction  workers.  2 

Immediately  after  construction  is  completed  an  effort  should  be  made  to  gather 
information  from  all  members  of  the  project  team  prior  to  its  dispersion.  The 
cataloging  of  this  information  can  provide  a  valuable  library  of  "lessons  learned"  and 
prevent  the  repetition  of  errors  on  future  projects. 

In  phase  three,  a  long  term  evaluation  needs  to  be  performed  to  evaluate  the 
overall  effectiveness  of  the  design.  Several  aspects  which  should  be  included  in  this 
evaluation  are  the  maintainability  of  the  facility,  layout  efficiency,  and  quality. 

Based  on  discussions  with  industry  personnel,  those  criteria  which  lend 
themselves  to  an  initial  design  evaluation  are:  Accuracy  of  the  Design  Documents, 
Usability  of  the  Design  Documents,  Cost  of  the  Design,  Constructability  of  the 
Design,  Economy  of  the  Design,  Performance  Against  Schedule,  and  Ease  of 
Occupancy.  These  criteria  are  discussed  in  the  following  paragraphs. 

The  most  obvious  products  of  the  design  process  are  the  drawings  and 
specifications.  The  accuracy  of  the  design  documents  criterion  evaluates  the 
effectiveness  of  the  design  by  monitoring  the  frequency  and  impact  of  errors  and 
omissions  in  the  drawings  and  specifications.  Examples  of  quantitative 
measurements  for  this  criterion  include  the  number  of  drawing  revisions,  the  number 
of  revised  drawings,  the  number  of  specification  revisions,  the  number  of  revised 
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specifications,  and  the  number  of  rework  manhours  due  to  design  errors  and 
omissions. 

The  usability  of  the  design  documents  measures  the  completeness,  clarity  and 
information  content  of  the  plans  and  specifications.  Subjective  measurements  for  this 
area  include  the  amount  of  cross  referencing,  the  appropriateness  of  the  drawing  size, 
the  completeness  of  the  drawings,  and  the  clarity  of  the  drawings. 

The  cost  of  the  design  effort  evaluates  the  anticipated  design  cost  versus  the 
actual  design  costs.  Caution  must  be  used  in  monitoring  this  criterion  as  many  factors 
may  impact  on  the  number  of  design  hours  and  cost  of  the  design  .  Owner  changes, 
regulatory  (code)  changes  and  schedule  compression  are  some  examples  of  factors 
that  may  impact  on  the  design  cost  and  duration.  These  factors  should  be  accounted 
for  through  a  revision  of  the  design  budget  and  schedule. 

Constructability  is  defined  as  the  optimum  integration  of  construction 
knowledge  and  experience  in  planning,  engineering,  procurement  and  field  operations 
to  achieve  overall  project  objectives.  3  General  areas  indicating  the  effectiveness  of  a 
project's  constructability  program  include  the  type  of  construction  equipment 
required,  size  and  skill  of  the  labor  force  required,  special  material  requirements,  and 
unrealistic  tolerances. 

The  economy  of  design  criterion  attempts  to  evaluate  the  efficiency  of  the  final 
design  product.  Indicators  of  an  efficient  design  include  minimizing  the  number  of 
overdesigned  structural  members  and  overspecified  materials,  and  the  efficiency  of 
the  building  layout. 

One  of  the  critical  factors  in  the  success  of  a  construction  project  is  the 
availability  of  the  design.     Delays  in  the  release  of  the  design  documents  can 
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adversely  affect  not  only  the  project's  overall  design  and  construction  schedule,  but 
may  increase  the  cost  of  bids  by  contractors,  vendors  and  material  suppliers. 

The  ease  of  occupancy  and  start-up  criterion  addresses  the  completeness  of  the 
design.  The  efficiency  of  the  design  process  is  judged  by  comparing  the  budgeted 
manhours  to  the  actual  manhours  required  to  prepare  the  facility  for  occupancy  or  use. 

These  seven  criteria  should  be  of  concern  to  each  of  the  design  users. 
Additionally,  with  only  limited  modifications  they  can  be  used  for  almost  any  type  of 
construction  project.  Before  their  final  selection  for  use  in  a  matrix,  each  criterion 
should  be  thoroughly  reviewed  not  only  for  its  applicability  to  type  of  project,  but 
also  to  determine  how  it  will  be  measured,  subjectively  or  quantitatively. 

3.3    Criteria  Weighting 

Each  of  these  criteria  will  vary  in  the  degree  of  their  accuracy  as  an  indicator 
of  design  effectiveness.  To  compensate  for  this,  each  criterion  is  weighted  by  the 
design  user  to  reflect  the  impact  of  the  criterion  in  relation  to  the  other  criteria.  The 
numt>er  of  points  assigned  to  a  criterion  is  a  direct  indication  of  its  influence  on  a 
design's  effectiveness. 

In  weighting  of  the  criteria  consideration  must  also  be  given  to  the  uniqueness 
of  the  individual  project.  A  project  that  is  schedule  driven  may  require  special 
emphasis  on  the  performance  against  schedule  criterion  and  constructability.  A 
project  that  is  on  an  extremely  tight  budget  may  place  additional  emphasis  on  the  cost 
of  the  design  and  economy  of  the  design. 

Finally,  the  objectiveness  or  subjectiveness  of  a  criterion  may  influence  its 
weight.    Due  to  a  possible  lack  of  confidence  in,  or  controversy  surrounding 
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subjective  ratings,  a  user  of  the  matrix  may  wish  to  emphasize  those  criteria  that  are 
measured  quantitatively. 
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Chapter  4 


APPLICATION  OF  THE  DESIGN  EVALUATION  MATRIX  TO 
COMMERCIAL  BUILDING  AND  HEAVY  CIVIL  PROJECTS 

To  explore  the  application  of  the  Design  Evaluation  Matrix  on  commercial 
buildings  and  heavy  civil  construction  projects,  it  was  necessary  to  obtain  data  from 
actual  projects  and  solicit  the  views  of  constructk>5ti  mdustry  professionals. 

Since  the  application  of  the  Evaluation  Matrix  is  relatively  new,  the  seven 
initial  criteria  for  measurement  of  design  effectiveness  were  used.  Industry  personnel 
were  encouraged  to  suggest  additional  criteria  and  sub-criteria  for  use  in  the 
evaluation  matrix. 

4.1     Methodology 

The  first  step  in  testing  the  feasibility  of  the  evaluation  matrix  was  the 
collection  of  project  data.  Questionnaires  were  mailed  out  to  CII  member  companies  / 
firms  requesting  projects  for  the  study. 

An  interview  guide  was  included  with  each  project  request  letter  to  aquaint 
personnel  designated  as  "points  of  contact"  with  the  purpose  of  the  study  prior  to 
interviews.  The  design  evaluation  matrix  was  presented  with  a  description  of  the 
criteria  and  sub-criteria.  Efforts  were  made  to  contact  owners,  designers,  and 
constructors  of  commercial  buildings  and  heavy  civil  construction  projects. 
However,  no  responses  received  from  design  firms. 

19 


mi 


Telephone  interviews  were  conducted  with  personnel  from  those  companies 
responding  to  the  request  for  projects.  The  projects  volunteered  for  the  study  ranged 
from  a  430,000  SF  office  complex  to  a  power  plant  facility  including  1500  feet  of 
tunnels  connecting  the  major  buildings.  All  project  personnel  interviewed  were 
degreed  engineers. 

The  interview  guide  was  divided  into  four  sections.  The  first  section  provided 
a  summary  of  the  research  project,  its  scope,  and  requested  a  point  of  contact.  An 
explanation  of  the  design  evaluation  matrix  was  presented,  criteria  and  sub-criteria 
discussed,  and  definitions  provided. 

Section  two  contained  a  project  description  sheet  requesting  general 
information  about  the  project.  Also  included  in  this  section  was  an  interview  guide 
addressing  each  of  the  seven  design  criteria,  and  requesting  the  information  required 
for  determining  the  sub-criteria  ratios. 

In  section  three  the  interviewees  were  asked  to  rank  the  criteria  and  sub- 
criteria  in  order  of  their  importance,  and  to  assign  weights  to  the  individual  criteria 
and  sub-criteria. 

In  section  four  the  interviewees  were  asked  to  estimate  the  industry  averages 
for  the  benchmark  performance  levels  of  ten,  three,  and  zero  for  each  of  the  sub- 
criteria.  These  are  subjective  estimates  based  on  the  cumulative  experience  and 
knowledge  of  the  interviewees. 

4.2     Sources  of  Data 

Letters  requesting  projects  for  the  study  were  sent  to  representatives  of 
owners,  designers,  and  constructors  belonging  to  the  Construction  Industry  Institute. 
Projects  were  requested  in  the  categories  of  commercial  buildings  (office  buildings. 
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corporate  headquarters  etc.)  and  heavy  civil  projects  (highway,  bridge,  and  the  heavy 
civil  phase  of  industrial   projects,  etc.). 

Responses  were  received  from  five  firms,  representing  owners  (3)  and 
constructors  (2).  No  response  was  received  from  a  design  firm.  One  constructor, 
who  was  also  the  facilitiy's  owner,  is  classified  as  an  owner.  The  projects  reviewed 
included  three  commercial  buildings  and  two  heavy  civil  projects.  Additionally,  one 
constructor  provided  assistance  in  the  determination  of  benchmark  values,  industry 
weights,  and  recommendations  for  additional  criteria  and  sub-criteria  in  the 
commercial  building  sector.  Figure  4. 1  provides  a  brief  description  of  each  project 
volunteered. 

Project  Summary 


Commercial  Buildings 

Classification 
Owner 


Owner 

Owner  (Constructor) 
Constructor 


FYoiect  Description 

Regional  headquarters  (300,(X)0SF),  offices, 

conference  facilities,  and  cafeteria. 

Regional  headquarters  (430,(XX)  SF), 
conference  facilities,  computer  center,  and 
cafeteria. 

Corporate  office  and  laboratory  expansion. 


Figure  4. 1 


Project  Summary  (Continued) 


Heavy  Civil 

Classification 
Constructor 


Constructor 


Project  Description 

Office  building,  refinery  power  plant,  facilities 

building,  1500  feet  of  interconnecting  tunnels. 

Nuclear  waste  processing  plant  (new 
technology) 


Figure  4. 1 


4.3     Design  Criteria 

The  initial  step  in  evaluating  design  effectiveness  is  the  identification  of  sub- 
criteria  for  each  of  the  design  criteria.  The  sub-criteria  should  be  much  more  specific 
that  the  criteria  on  the  master  matrix. 

In  an  effort  to  increase  the  acceptance  of  the  evaluation  matrix  sub-criterion 
should  be  quantitatively  measured.  Subjective  measurements  should  be  used  only 
when  absolutely  necessary. 

The  sub-criterion  for  each  of  the  seven  criteria  are  shown  in  Figure  4.2: 
Criteria  and  Sub-Criteria.  The  abbreviated  criteria  and  sub-criteria  tides  utilized  in  the 
matricies  are  contained  in  parentheses. 
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Criteria  and  Sub-Criteria 


ACCURACY  OF  THE  DESIGN  DOCUMENTS  (ACCURACY) 

-  Drawing  revisions  /  total  number  of  drawings  (Drawing  Revisions) 

-  Revised  drawings  /  total  number  of  drawings  (Revised  Drawings) 

-  Specification  revisions  /  total  number  of  specifications  (Specification  Revisions) 

-  Revised  Specifications  /  total  number  of  specifications  (Revised  Specifications) 

-  Manhours  of  rework  /  total  number  of  manhours  (Rework) 

USABILITY  OF  THE  DESIGN  DOCUMENTS  (USABILITY) 

-  Subjective  rating  for  drawing  size  (Drawing  Size) 

-  Subjective  rating  for  number  of  drawings  (Number  of  Drawings) 

-  Subjective  rating  for  number  of  cross  references  (Cross  References) 

-  Field  engineering  manhours  /  total  engineering  manhours  (Field  Engineering) 

-  Subjective  rating  for  clarity  of  drawings  (Clarity) 

-  Subjective  Rating  for  completeness  of  drawings  (Completeness) 

COST  OF  DESIGN  EFFORT  (COST) 

-  Design  cost  /  construction  cost  (Design  Cost) 

-  Design  manhours  expended  /  design  manhours  budgeted  (Design  Manhours) 

CONSTRUCTABILITY  (CONSTRUCTABILITY) 

-  Subjective  rating  for  the  number  of  unrealistic  tolerances  vs.  quality  expectations 
(Tolerances) 

-  Subjective  rating  for  the  number  of  crafts  (Different  Crafts) 

-  Subjective  rating  for  compatibihty  with  current  materials  and  technology 
(Compatibility) 

ECONOMY  OF  DESIGN  (ECONOMY) 

-  Building  layout  efficiency  (Layout) 

-  Amount  of  overdesigned  members  (Overdesign) 

-  Amount  of  overspecified  materials  (Overspecified) 

PERFORMANCE  AGAINST  SCHEDULE  (SCHEDULE) 

-  Percent  of  construction  document  release  dates  attained  (Document  Release) 

-  Percent  of  intermediate  release  dates  attained  (Intermediate  Release) 

EASE  OF  OCCUPANCY  (OCCUPANCY) 

-  Occupancy  preparation  days  required  /  days  budgeted  (Occupancy  Days) 

-  Number  of  maintenance  personnel  required  /  number  budgeted  (Personnel) 


Figure  4.2 


The  accuracy  of  the  design  dcx:uments  criterion  concerns  the  number  of  errors 
and  omissions  in  the  plans  and  specifications.  For  commercial  buildings  and  heavy 
civil  projects,  as  with  most  construction  projects,  accuracy  is  best  reflected  by  the 
number  of  revisions  in  the  design  documents,  and  the  number  of  rework  manhours 
due  to  these  errors  and  omissions.  In  evaluating  the  accuracy  of  the  design 
documents  only  those  drawings  issued  for  construction  are  reviewed.  Prior  to 
issuance,  a  drawing  is  not  considered  to  be  complete.  Sub-criteria  utilized  in 
evaluating  the  accuracy  criteria  include  distinctions  between  the  number  of  drawing 
revisions  and  the  number  of  revised  drawings,  and  the  number  of  specification 
revisions  and  the  number  of  revised  specifications.  This  was  necessary  since  the 
impact  of  numerous  revisions  to  a  single  drawing  or  specification  may  result  in  a 
different  impact  when  compared  to  a  larger  number  of  drawings  or  specifications 
being  revised.  '  " 

Usability  of  the  design  documents  is  primarily  a  subjective  criterion  that 
reflects  the  information  content  and  format  of  the  drawings  and  specifications.  The 
use  of  subjective  ratings  in  this  and  other  criteria  must  be  approached  with  caution  due 
to  the  different  education,  experiences,  and  standards  of  each  evaluator.  The  only 
quantitative  sub-criteria  included  in  this  evaluation  is  the  number  of  field  engineering 
manhours  versus  the  total  number  of  design  manhours. 

The  cost  of  the  design  can  be  subdivided  for  the  various  phases  (electrical, 
mechanical,  HVAC,  etc.)  of  a  project  to  determine  the  overall  cost  effectiveness  of 
the  project.  The  actual  cost  versus  the  budgeted  costs  is  one  measure  of  the 
effectiveness  of  the  design  effort.  Due  to  the  preliminary  nature  of  this  study  no 
subdivisions  were  made.  Allowances  should  also  be  made  for  owner  initiated 
changes  affecting  scope,  installed  equipment,  materials,  and  methods. 
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The  constructability  criterion  is  subjectively  rated  and  difficult  to  measure  due 
to  the  unique  aspects  of  each  project's  program.  Some  factors  that  impact  on 
constructability  include  unrealistic  tolerances  which  can  increase  costs  and  slow 
production,  and  the  type  of  equipment  to  b>e  installed  which  when  coupled  with  the 
number  of  different  crafts  required  may  increase  the  probability  of  jurisdictional 
disputes  on  union  projects.  Other  important  items  to  be  reviewed  include  the  use  of 
prefabrication  yards,  models,  taking  advantage  of  repetitive  operations  to  obtain  the 
maximum  benefits  from  learning  curves,  and  the  necessity  of  specialized  equipment 
and  skills. 

The  economy  of  design  criterion  is  indicative  of  a  design's  waste  and 
inefficiency.  Material  waste  adds  directiy  to  the  cost  of  a  project  and  operational 
inefficiencies  may  create  scheduling  and  access  problems.  Additionally,  the  presence 
of  overdesigned  members  and  overspecified  materials  can  increase  the  cost  of  a 
project. 

For  a  project  to  remain  on  schedule  the  design  documents  must  be  issued  on 
their  scheduled  release  dates.  Poor  performance  in  the  release  of  plans  and 
specifications  can  impact  on  areas  such  as  procurement,  quality,  and  schedule. 

The  ease  of  occupancy  and  start-up  criterion  attempts  to  evaluate  the 
completeness  of  the  design  and  identify  unexpected  problem  areas.  An  increase  in  the 
maintenance  manhours  required  prior  to  occupancy  or  use  may  be  indicative  of  a 
design  omission. 

During  the  interviews  industry  personnel  were  asked  to  suggest  additional 
criteria  and  sub-criteria  to  add  to  the  matrix.  Suggested  additions  to  the  seven  criteria 
for  the  master  matrix  are: 

1 .   Aesthetics 


2.  Maintainability 

3.  Installed  systems  reliability. 

4.  Expandability 

5.  Quality 

6.  Durability  of  finishes 

7.  Safety 

Recommendations  for  additional  sub-criteria  included: 
Accuracy  of  the  Design  Documents 

1 .  The  number  of  drawing  clarification  memos  sent  to  the  designer  by  the 
contractor  /  the  total  number  drawings. 

2.  Number  of  changes  due  to  misunderstanding  of  the  building  code 
requirements. 

3.  Number  of  changes  due  to  errors  and  omissions  and  their  associated 
costs. 

Usability  of  the  Design  Documents 

1.  The  number  of  requests  for  information  sent  to  the  designer  by  the 
contractor. 

2.  Number  of  changes  due  to  shop  drawing  review  comments. 
Cost  of  the  Design  Effort 

1 .  Duration  of  the  design. 
Constructability 

1 .  Engagement  of  the  contractor. 


Maintainability 

1.  Accessibility  of  building  utilities  /  services    (e.g.  plumbing,  HVAC, 
piping,  ducts,  controls,  etc.)  for  maintenance  and  repair. 

2.  Maintenance  manhours  budgeted  vs.  maintenance  manhours  expended. 

3.  Actual  cost  of  maintenance  vs.  budgeted  costs. 

4.  Quality  of  documentation  (as-buUt  drawings). 

5.  Durability  of  materials  chosen  for  high  traffic  areas. 

4.4     Weights 

The  evaluation  criteria  used  in  the  Design  Evaluation  Matrix  were  assigned 
weights  to  reflect  their  impact  on  the  effectiveness  of  design.  These  weights,  based 
on  the  averages  of  the  weights  assigned  by  industry  personnel,  are  shown  in  Figure 
4.3:  Criteria  Weights. 


Criteria  Weights 


Criteria 

Accuracy 

Usability 

Cost 

ConstructabLlity 

Economy 

Schedule 

Occupancy 


Commercial  Building 
20 
20 
10 
20 
10 
15 


Heavy  Civil 
20 
20 
10 
25 
10 
10 
5 


Figure  4.3 
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For  commercial  buildings  and  heavy  civil  projects,  the  criteria  accuracy  of  the 
design  documents,  usability  of  the  design  documents,  and  constructability,  were 
ranked  as  the  most  important  indicators  of  design  effectiveness.  A  slighdy  higher 
weight  was  assigned  to  constructability  by  the  heavy  civil  evaluators  than  by  those 
evaluating  commercial  buildings.  While  assigned  equal  weights,  the  top  three 
evaluation  criteria  for  commercial  buildings  in  rank  order,  were,  constructability, 
accuracy  of  the  design  documents,  and  usabiUty  of  the  design  documents. 

The  sub-criteria  for  each  criterion  were  also  assigned  weights  to  reflect  their 
relationship  to  the  evaluation  criterion  on  the  master  matrix.  These  assignments  are 
presented  in  Figures  4,4  through  4.10. 

On  both  commercial  building  and  heavy  civil  projects,  the  amount  of  rework 
was  considered  to  be  the  most  important  indicator  of  design  document  accuracy.  This 
was  expected  due  to  the  relationships  between  rework,  cost,  productivity,  schedule, 
and  worker  morale.  The  number  of  revised  specifications  was  rated  the  least 
important  sub-criteria  for  heavy  civil  projects. 

Accuracy  Sub-Criteria  Weights 


Criteria 

Commercial  Building 

Heaw  Civil 

Drawing  Revisions 

15 

15 

Revised  Drawings 

20 

15 

Specification  Revisions 

15 

20 

Revised  Specifications 

15 

10 

Rework 

35 

40 

Figure  4.4 
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The  usability  of  the  design  dcx^uments  was  ranked  as  the  third  most  important 
evaluation  criteria  on  both  commercial  building  and  heavy  civil  projects.  The  sub- 
criteria,  completeness,  and  clarity,  were  judged  the  most  important  indicators  of 
design  document  usability  for  both  types  of  construction. 

Usability  Sub-Criteria  Weights 


Criteria 

Commercial  Building 

Heaw  Civil 

Drawing  Size 

5 

5 

Number  of  Drawings 

10 

5 

Cross  References 

15 

10 

Field  Engineering 

15 

10 

Qarity 

25 

35 

Completeness 

30 

35 

Figure  4.5 


The  cost  of  the  design  criterion  was  generally  ranked  low  in  importance.  Of 
the  sub-criteria,  personnel  involved  with  heavy  civil  projects  favored  design 
manhours  over  design  cost.  One  interesting  response  was  the  assignment  of  an 
extremely  high  weight  to  a  write-in  criteria,  duration  of  the  design  process,  by  one  of 
the  evaluators  for  commercial  buildings. 
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Cost  Sub-Criteria  Weights 


30 


Criteria 
Design  Cost 
Design  Manhours 


Commercial  Building 
75 
25 


Heavy  Civil 
40 
60 


Figure  4.6 

Overall,  the  constructability  criterion  was  rated  as  the  most  important  indicator 
of  design  effectiveness.  In  evaluating  the  constructability  sub-criteria,  both  industry 
segments  rated  the  amount  of  unrealistic  tolerances  as  the  most  important  factor. 
Compatibility  of  the  design  with  current  materials  and  technology  was  ranked  second 
by  both  groups. 

Constructability  Sub-Criteria  Weights 


Criteria 
Tolerances 
Different  Crafts 
Compatibility 


Commercial  Building 
50 
10 
40 


Heaw  Civil 
60 
10 
30 


Figure  4.7 
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Economy  Sub-Criteria  Weights 


Criteria 
Layout 
Overdesign 
Overspecified 


Commercial  Building 
55 
25 
20 


Heavy  Civil 
15 
45 
40 


Figure  4.8 

For  commercial  building  projects  the  sub-criteria  layout  was  favored  by  a 
wide  margin.  It  is  interesting  to  note  that  both  overdesigned  members  and 
overspecified  materials  were  weighted  very  close  together  for  heavy  civil  projects, 
while  layout  received  relatively  little  attention. 

Schedule  Sub-Criteria  Weights 


Criteria 

Document  Release 
Intermediate  Release 


Commercial  Building 
55 
45 


Heavy  Civil 
45 
55 


Figure  4.9 


For  the  performance  against  schedule  criterion,  both  sub-criteria  were 
regarded  as  relatively  equal.  Those  personnel  evaluating  commercial  buildings 
favored  document  release  dates  met,  while  the  heavy  civil  evaluators  favored 
intermediate  document  release  dates. 


Occupancy  Sub-Criteria  Weights 


Criteria  Commercial  Building  Heavy  Civil 

Occupancy  55  55 

Personnel  45  45 


Figure  4.10 

The  occupancy  /  start-up  criterion  was  ranked  the  least  important  on  both 
commercial  and  heavy  civil  projects.  Weight  assignments  for  the  sub-criteria  were 
identical  with  the  number  of  days  required  to  prepare  for  occupancy  /  put  the  facility 
into  use  favored  over  the  number  of  manhours  required  to  prepare  the  building  / 
facility. 

4.5     Performance  Scale  Assignments 

The  assignment  of  benchmark  values  by  industry  personnel  are  shown  in 
Figures  4.11  and  4.12,  Estimated  Benchmark  Values.  These  are  the  values  that 
would  be  entered  into  the  master  and  sub-matricies  at  the  performance  levels  of  ten, 
three,  and  zero.  The  increments  between  these  scores  would  be  determined  by 
utilizing  the  percentage  increment  technique,  numerical  increment  technique,  or  the 
subjective  ratio  technique.  Caution  must  be  used  in  applying  these  values  to  actual 
projects  due  to  the  limited  number  of  projects  studied  during  this  investigation. 

Figure  4.11  presents  the  estimated  benchmark  values  for  commercial 
buildings  based  on  the  projects  volunteered  for  the  study.  The  "Measured  Value" 
represents  the  average  of  the  values  assigned  to  a  particular  sub-criteria  by  industry 
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Estimated  Benchmark  Values 
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Figure  4. 11 

personnel  for  the  projects  volunteered  for  the  study.  All  values  are  expressed  as 
percentages.  A  comparison  of  the  measured  values  with  the  estimated  values 
indicates  that  the  average  project  is  generally  between  the  level  three  and  level  zero 
benchmarks.  This  is  even  more  apparent  when  the  measured  values  are  placed  in  the 
example  evaluation  matrix  in  the  next  chapter. 
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Figure  4.12  presents  the  estimated  benchmark  values  for  heavy  civil  projects 
based  on  the  projects  studied.  A  comparison  of  project  measured  values  with  the 
estimated  benchmark  values  indicates  that  the  average  heavy  civil  project  falls  between 
the  level  three  and  level  zero  benchmarks.  The  measured  values  for  two  of  the  sub- 
Estimated  Benchmark  Values 
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criteria,  specification  revisions  and  revised  specifications  score  substantially  above 
the  zero  level  of  performance.  These  values  are  heavily  influenced  by  the  new, 
unproven  technology  associated  with  one  of  the  heavy  civil  projects  studied. 
Sufficient  data  is  not  currently  available  to  determine  the  actual  values  that  should  be 
used  for  the  benchmark  scores.  These  values  are  provided  for  illustrative  purposes 
only. 

4.6     Conclusions 

The  Design  Evaluation  Matrix  is  extremely  flexible  and  holds  a  great  deal  of 
promise  for  use  throughout  the  construction  industry.  The  information  required  to 
construct  the  matricies  is  readily  available  on  current  construction  projects.  Due  to  the 
limited  number  of  projects  investigated  during  this  study,  the  values  presented  in  this 
paper  should  be  considered  for  illustrative  purposes  only.  As  the  evaluation  matrix 
receives  more  attention  from  industry  the  database  for  criteria  and  sub-criteria  weights 
and  benchmark  levels  should  expand  and  become  more  refined. 


Chapter  5 


DESIGN  EVALUATION  MATRIX  EXAMPLE 

The  next  step  in  exploring  the  application  of  the  Design  Evaluation  Matrix  to 
the  areas  of  commercial  buildings  and  heavy  civil  construction  is  the  construction  of 
an  example  matrix.  In  this  chapter,  an  evaluation  matrix  for  a  commercial  building 
will  be  constructed  and  discussed. 

5.1     Construction  of  the  Matricies 

The  first  step  in  the  construction  of  the  master  matrix  and  sub-matricies  is 
entering  the  criteria  and  sub-criteria  column  headings,  the  weight  assignments,  and 
the  level  zero,  three,  and  ten  benchmark  estimates  in  the  matrix  forms.  For  each 
criterion  in  the  master  matrix  and  sub-criterion  in  the  sub-matricies,  the  benchmark 
values  collected  during  the  interviews  will  be  used. 

After  all  the  interview  data  has  been  entered  in  the  matricies  scores  must  be 
assigned  to  the  increments  between  the  benchmark  levels  of  zero,  three,  and  ten.  In 
this  example,  the  percentage  increment  technique  will  be  utilized  to  determine  the 
scores  for  the  sub-criteria  rework,  design  cost,  design  manhours,  document  release, 
intermediate  release,  and  occupancy.  The  sub-criteria  scores  for  drawing  revisions, 
revised  drawings,  specification  revisions,  field  engineering,  and  maintenance 
manhours  will  be  determined  using  the  numerical  increment  method.  Subjectively 
rated  sub-criteria  scales  are  equal  to  their  scores. 
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In  order  to  determine  the  performance  index  for  each  of  the  evaluation  criteria 
on  the  master  matrix  the  sub-matricies  must  first  be  completed.  The  performance 
scores,  representing  the  averages  of  the  data  collected  during  the  study,  are  entered  in 
the  row  labeled  performance  immediately  below  the  sub-criteria  headings.  The  score 
closest  to  these  values  in  the  sub-matrix  body  are  circled  and  recorded  in  the  row 
marked  "score"  at  the  bottom  of  the  matrix  form.  Scores  are  multiplied  by  their 
corresponding  weights,  and  the  results  entered  in  the  row  labeled  "value".  The  values 
for  each  sub-criteria  are  added  to  yield  the  "performance  index". 

The  sub-criteria  performance  indicies  are  transferred  to  the  master  matrix  form 
and  recorded  under  the  appropriate  criteria  heading  in  the  row  marked  "performance". 

The  score  that  is  equal  to  or  less  than  the  performance  score  is  circled  in  the 
body  of  the  matrix  and  entered  at  the  bottom  of  the  matrix  in  the  row  labeled  "score". 
These  scores  are  multiplied  by  their  corresponding  weights  and  the  results  are 
recorded  in  the  "value"  row.  The  values  are  added  together  to  produce  the  overall 
"performance  index". 

5.2    Matricies  Analysis 

The  completed  sub-criteria  evaluation  matricies  and  the  master  matrix  for  the 
example  are  presented  in  Figures  5.1  through  5.8.  Each  will  be  discussed  in  the 
following  paragraphs. 

The  sub-matrix  for  Accuracy  of  the  Design  Documents  is  shown  in  Figure 
5.1.  Accuracy  of  the  Design  Documents  was  ranked  as  the  second  most  important 
criterion.  Based  on  the  data  utilized  for  this  example,  the  sub-matrix  has  a 
performance  index  of  465.  While  this  index  is  above  average,  the  majority  of  its 
value  comes  from  two  sub-criteria,  rework,  and  revised  specifications.    Thus, 


attention  should  be  directed  to  the  sub-criteria  revised  drawings,  drawing  revisions, 
and  specification  revisions.  Each  of  these  sub-criteria  have  extremely  low  scores, 
with  the  sub-criteria,  revised  drawings,  contributing  zero  points  to  the  index. 

Figure  5.2  is  the  Usability  of  the  Design  Documents  Sub-Matrix.  With  a 
performance  index  of  240,  this  sub-matrix  is  considered  below  average  (300).  Two 
of  the  six  sub-criteria  were  rated  average,  and  one,  number  of  drawings,  was  rated 
above  average.  However,  the  most  important  sub-criterion,  completeness  of  the 
design  documents,  received  the  lowest  actual  score.  This  tends  to  reemphasize  the 
problems  identified  in  the  Accuracy  of  the  Design  Documents  Sub-Matrix  conceming 
the  number  of  drawings  and  specification  revisions,  and  the  number  of  revised 
drawings.  Another  interesting  observation  in  this  sub-matrix  concerns  the  scores 
associated  with  those  sub-criteria  which  were  rated  subjectively.  In  previous 
research,  subjectively  rated  criteria  consistently  received  scores  higher  than 
quantitatively  rated  sub-criteria.  This  trend  was  not  encountered  during  this 
investigation. 

The  Cost  of  the  Design  Effort  Sub-Matrix  is  shown  in  Figure  5.3.  The  cost 
of  the  design  effort  was  given  the  second  lowest  priority  by  industry  personnel.  With 
a  combined  score  of  100,  it  has  the  second  lowest  performance  index. 

Constructability  was  ranked  as  the  most  important  design  evaluation  criterion. 
The  Constructability  Sub-Matrix  is  presented  in  Figure  5.4.  As  the  most  important 
criterion,  the  sub- matrix's  performance  index  of  1 80,  well  below  average,  should  be 
of  major  concern.  The  most  important  sub-criteria,  unrealistic  tolerances  versus 
quality  expectations,  has  tied  for  the  lowest  score.  Though  this  study  was  limited  in 
the  number  of  projects  investigated,  this  particular  sub-matrix  illustrates  the 
importance  and  potential  of  constructability  programs. 
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Figure  5.5  is  the  Economy  of  Design  Sub-Matrix.  One  interesting  aspect  of 
this  sub-matrix  is  the  low  score  of  the  layout  sub-criteria.  As  mentioned  earlier,  no 
responses  were  received  from  design  firms.  The  performance  index  of  345  is  slightly 
above  average. 

Figure  5.6  presents  the  Performance  Against  Schedule  Sub-Matrix.  All 
contributors  to  the  study  were  unhappy  with  the  schedule  performance  of  the  design 
documents.  Comments  form  one  constructor  indicated  that  each  time  a  scheduled 
release  or  intermediate  release  date  was  missed,  it  was  simply  rescheduled  as  if 
nothing  had  happened.  The  performance  index  of  0  is  the  lowest  of  any  sub-matrix. 

The  Ease  of  Occupancy  /  Start-up  Sub-Matrix  is  presented  in  Figure  5.7.  This 
criterion  was  assigned  the  lowest  weight  by  industry  evaluators.  The  performance 
index  for  this  sub-matrix  was  495,  slightly  above  average.  Owners  generally  were 
able  to  occupy  their  projects  on  or  slightly  ahead  of  schedule,  but  required  extra 
personnel  to  prepare  the  facilities. 

Figure  5.8,  The  Design  Evaluation  Matrix,  presents  a  completed  master 
matrix  based  on  the  values  (performance  indicies)  obtained  from  the  sub-matricies. 
The  overall  performance  index  for  this  example  is  200,  below  average.  While  a 
below  average  index  is  disappointing,  the  example  illustrates  the  usefulness  of  the 
method  in  identifying  problem  areas.  Attention  should  be  directed  to  those  criteria 
which  scored  below  in  their  respective  sub-matricies,  usability  of  the  design 
documents,  cost  of  the  design  effort,  constructability,  and  performance  against 
schedule.  Additionally,  due  to  the  influence  of  the  accuracy  of  the  design  documents 
on  these  criteria,  efforts  should  be  made  to  improve  the  performance  of  this  criterion. 


t>r 


40 


Vi 

C 

CW 

o 

c 

Vi 

t« 

o 

bO 

•o 

> 

c 
1 

05 

s 

IS 

^ 

j5 

c 
o 

^, 

to 

e 

1 

> 

'a 

s 

i3 

C/3 

1 

•g 

1 

a: 

00 

02 

u 

Oi 

35 

46 

40 

9 

2.3 

5 

5 

5 

5 

2 

8 

8 

8 

7.8 

C"]: 

11 

11 

11 

s'O 

2.6 

14 

14 

14 

14 

3 

16 

16 

16 

16 

3.3 

19 

19 

19 

19 

3.8 

22 

22 

22 

22 

4.4 

25 

25 

25 

25 

5 

33 

33 

33 

33 

7 

:^^: 

42 

:«;; 

42 

10 

50 

S-: 

50 

50 

15 

PERFORMANCE 


10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


1 

0 

i 

8 

9 

1 

SCORE 

15 

20 

15 

15 

35 

WEIGHT 

15 

0 

15 

120 

315 

VALUE 

PERFORMANCE 
INDEX 


Accuracy  of  the  Design  Documents  Sub-Matrix 


FIGURE   5.1 


.■^^ 


41 


e 


o 


.1 

c 


c 

a: 

e 

u 


bO 


c 


35 


U 


Si 
o 


U 


PERFORMANCE 


10 

10 

10 

5 

10 

10 

9 

9 

9 

9 

9 

9 

8 

8 

12 

8 

8 

7 

7 

7 

16 

7 

7 

6 

6 

6 

19 

6 

6 

5 

5 

5 

23 

5 

5 

4 

4 

4 

26 

4 

4 

3 

3 

CO 

30 

CO 

3 

2 

sO 

2 

CO 

2 

2 

1 

1 

1 

47 

1 

CO 

0 

0 

0 

55 

0 

0 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


8 

2 

3 

2 

3 

1 

SCORE 

5 

10 

15 

15 

25 

30 

WEIGHT 

40 

20 

45 

30 

75 

30 

VALUE 

PERFORMANCE 
INDEX 


Usability  of  the  Design  Documents  Sub-Matrix 


FIGURE   5.2 


42 


o 
U 

& 
Q 


& 


116 


PERFORMANCE 


4 

100 

4.3 

103 

4.7 

105 

5.1 

108 

5.5 

111 

6 

114 

6.5 

:"i 

7 

120 

7.6 

126 

8.3 

133 

^O 

140 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


0 

4 

SCORE 

75 

25 

WEIGHT 

0 

100 

VALUE 

PERFORMANCE 
INDEX 


0^ 


Cost  of  Design  Effort  Sub-Matrix 


FIGURE   5.3 


43 


>, 

u 

x> 

(3 

■^^ 

•3 

u 

C 

§, 

g 

!S 

o 

e2 

Q 

U 

1 

1 

3 

PERFORMANCE 


10 

10 

10 

9 

9 

9 

8 

8 

8 

7 

7 

7 

6 

6 

6 

5 

5 

5 

4 

4 

4 

3 

3 

sO 

2 

2 

2 

CO 

sO 

1 

0 

0 

0 

r 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


1 

1 

3 

SCORE 

50 

10 

40 

WEIGHT 

50 

10 

120 

VALUE 

PERFORMANCE 
INDEX 


dfD 


Constructability  Sub-Matrix 


FIGURE   5.4 


44 


^ 

S) 

iS 

a 

1 

^ 

P 

^ 

& 

2 

o 

5 

2 

3 

8 

PERFORMANCE 


10 

10 

10 

9 

9 

9 

8 

8 

sO 

7 

7 

7 

6 

6 

6 

5 

5 

5 

4 

4 

4 

3 

CO 

3 

:o 

2 

2 

1 

1 

1 

0 

0 

0 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


2 

3 

8 

SCORE 

55 

25 

20 

WEIGHT 

110 

75 

160 

VALUE 

PERFORMANCE 
INDEX 


Economy  of  Design  Sub-Matrix 


FIGURE  5.5 


i4- 


45 


68 


DC 


B 

S 


64 


PERFORMANCE 


100 

100 

97 

95 

94 

90 

91 

86 

88 

82 

85 

78 

83 

74 

80 

80 

75 

77 

70 

73 

:«:; 

» 

r 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


0 

0 

SCORE 

55 

45 

WEIGHT 

0 

0 

VALUE 

PERFORMANCE 
INDEX 


Performance  Against  Schedule  Sub-Matrix 


FIGURE   5.6 


46 


>-. 

u 

c 
c 

p 

o 

^ 

S2 

92 

127 

PERFORMANCE 


90 

90 

^C 

93 

95 

97 

98 

100 

101 

104 

104 

107 

107 

HI 

110 

115 

116 

118 

123 

125 

130 

::,3& 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


9 

0 

SCORE 

55 

45 

WEIGHT 

495 

0 

VALUE 

PERFORMANCE 
INDEX 


Ease  of  Occupancy  /  Start-up  Sub-Matrix 


FIGURE   5.7 


47 


t 

d 

§ 

H 

>- 

< 

8 

< 

D 

H 
O 

U 
D 

s 

Q 

S 

u 

C/3 

465 

240 

100 

180 

345 

0 

495 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

900 

900 

900 

900 

900 

900 

900 

800 

800 

800 

800 

800 

800 

800 

700 

700 

700 

700 

700 

700 

700 

600 

600 

600 

600 

600 

600 

600 

500 

500 

500 

500 

500 

500 

500 

^0^ 

400 

400 

400 

400 

400 

^0& 

300 

300 

300 

300 

S^^/ 

300 

300 

200 

>^ 

200 

200 

200 

200 

200 

100 

100 

0^ 

0^/ 

100 

100 

100 

0 

0 

0 

0 

0 

s  «^ 

0 

PERFORMANCE 


10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 


SCORE 


4 

2 

1 

1 

3 

0 

4 

SCORE 

20 

20 

10 

20 

10 

15 

5 

WEIGHT 

80 

40 

10 

20 

30 

0 

20 

VALUE 

PERFORMANCE 
INDEX 


Design  Evaluation  Matrix 


FIGURE   5.8 


48 


5.3     Conclusions 

Based  on  this  example  matrix  evaluation,  several  conclusions  can  be  reached. 
The  data  required  to  evaluate  design  effectiveness  on  both  commercial  building  and 
heavy  civil  construction  projects  is  readily  available.  While  most  people  interviewed 
felt  more  comfortable  with  quantitative  measurements,  no  identifiable  score  inflation 
occurred  due  to  the  use  of  subjective  ratings.  The  application  of  the  matrix  to  an 
entire  project  is  rather  simplistic.  A  more  extensive  study  will  be  required  to 
determine  the  actual  weights  and  benchmark  values  that  should  be  assigned  to  the 
various  criteria  and  sub-criteria.  Each  project  should  be  sub-divided  into  the  major 
design  divisions  for  a  detailed  analysis.  All  users  of  the  design,  owners,  designers, 
and  constructors  need  to  be  included  in  the  collection  of  data  for  a  realistic  database. 
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Chapter  6 


CONCLUSIONS  AND  RECOMMENDATIONS 

This  report  investigates  the  feasibility  of  utilizing  a  matrix  evaluation  technique 
known  as  the  Design  Evaluation  Matrix,  as  a  method  to  evaluate  the  design 
effectiveness  in  commercial  building  and  heavy  civil  construction.  The  Design 
Evaluation  Matrix,  a  variation  of  the  Objectives  Matrix,  has  proven  to  be  a  flexible 
and  effective  method  to  categorize  and  weight  those  factors  that  have  a  significant 
impact  on  a  construction  project.  This  final  chapter  presents  the  conclusions  of  the 
study  and  recommendations  for  future  research. 

6.1      Conclusions 

1 .  The  seven  design  evaluation  criteria  identified  in  the  initial  phase  of  the 
research  are  applicable  to  all  construction  projects  regardless  of  industry 
segment. 

2.  The  Design  Evaluation  Matrix  can  be  used  during  the  individual  design, 
construction,  and  operation  phases  of  a  project,  or  combined  for  a  more 
simplistic  evaluation  of  an  entire  project. 
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3.  The  list  of  evaluation  criteria  should  be  expanded.  This  will  occur  as  a 
by-product  of  the  method's  utilization  by  industry,  and  feedback  being 
provided  to  researchers. 

4.  The  data  collected  for  this  study  was  inconclusive  (due  to  the  limited  data 
available)  for  determining  the  values  that  should  be  used  by  industry  for 
weighting  the  criteria  (sub-criteria)  and  establishing  benchmark  values. 

6.2     Recommendations 

1.  An  extensive  database  should  be  established  for  each  segment  of  the 
construction  industry,  reflecting  the  views  of  owners,  designers,  and 
constructors. 

2.  Data  collections  should  be  undertaken  to  establish  benchmark  values  for 
each  major  design  function  (electrical,  mechanical,  HVAC,  et  c). 

3.  As  the  method  is  adopted  for  use  by  industry,  feedback  should  be 
collected  to  further  refine  the  database. 

4.  The  experiences  of  companies  utilizing  the  method  need  to  be  cataloged  in 
a  "lessons  learned"  format  in  order  to  periodically  review  the  guidelines 
for  applying  the  method. 
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5.  The  additional  design  evaluation  criteria  and  sub-criteria  summarized  in 
Figure  6.1  should  be  considered  for  future  evaluations  of  commercial 
building  and  heavy  civil  projects. 
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Additional  Criteria  and  Sub-Criteria 


Evaluation  Criteria 

1. 

Aesthetics 

2. 

Maintainability 

3. 

Installed  systems  reliability 

4. 

Expandability 

5. 

Quality 

6. 

Durability  of  finishes 

7. 

Safety 

Sub-Criteria 

Accuracy  of  the  Design  Documents 

1 .  The  number  of  drawing  clarification  memos  sent  to  the  designer  by  the 
contractor  the  total  number  drawings. 

2.  Number  of  changes  due  to  misunderstanding  of  the  building  code 
requirements. 

3.  Number  of  changes  due  to  errors  and  omissions  and  their  associated 
costs. 

Usability  of  the  Design  Documents 

1 .  The  number  of  requests  for  information  sent  to  the  designer  by  the 
contractor. 

2.  Number  of  changes  due  to  shop  drawing  review  comments. 

Cost  of  the  Design  Effort 

1 .  Duration  of  the  design. 

Constructability 

1.  Engagement  of  the  contractor. 

Maintainability 

1 .  Accessibility  of  building  utiUties  /  services  (e.g.  plumbing,  HVAC, 
piping,  ducts,  controls,  etc.)  for  maintenance  and  repair. 

2.  Maintenance  manhours  budgeted  vs.  maintenance  manhours  expended. 

3.  Actual  cost  of  maintenance  vs.  budgeted  costs. 

4.  Quality  of  documentation  (as-built  drawings). 

5.  Durability  of  materials  chosen  for  high  traffic  areas. 


Figure  6. 1 


APPENDIX  1 

INCREMENT  FORMULA  TECHNIQUE 
PERCENT  FORMULA  TECHNIQUE 
SUBJECTIVE  RATIO  TECHNIQUE 
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AU Nunrgncal  Incrcingnt  Tgchnique 

The  following  formula  can  be  utilized  to  determine  the  numerical  increment 
between  benchmark  levels: 

Numerical  Increment  Formula 

(X-Y)/Z  =  I 

X  -  represents  the  larger  benchmark  score  ratio 

Y  -  represents  the  smaller  benchmark  score  ratio 

Z  -  represents  the  number  of  increments  between  the  benchmark  scores,  either 

7  (between  levels  three  and  ten)  or  3  (between  levels  zero  and  three). 
I  -  represents  the  numerical  ratio  increment  between  two  consecutive  levels 
going 

from  Y  to  X. 


.  ,.J 


.-nl 
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A1.2     Percentage  Increment  Technique 

The  following  formula  can  be  utilized  to  determine  the  percentage  increment 
between  benchmark  levels: 

Percentage  Increment  Formula 

(Y/X)l/z  =  K 

X  -  represents  the  larger  benchmark  score  ratio 

Y  -  represents  the  smaller  benchmark  score  ratio 

Z  -  represents  the  number  of  increments  between  the  benchmark  scores,  either 

7  (between  benchmark  levels  three  and  ten)  or  3  (between  benchmark  levels 

zero  and  three). 
K  -  represents  the  percentage  decrease  of  ratios  between  consecutive  levels 

going 

from  X  to  Y. 


>? 
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A1.3     Subjective  Ratio  Technique 

This  technique  involves  the  use  of  subjective  ratios  to  assign  values  to  the 
levels  between  the  benchmark  levels.  This  type  of  ratio  assignment  requires 
personnel  knowledgeable  with  the  criteria,  objectives,  and  workers.  Concurrence  by 
those  being  evaluated  is  desired  to  ensure  the  maximum  cooperation  and 
understanding  of  the  workforce. 


'     »  ■». 


APPENDIX  2 


PROJECT  REQUEST  LETTER 
INTERVIEW  GUIDE 
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Project  Request  Letter 


* 

Dear  : 

Enclosed  is  an  explanation  of  how  the  Construction  Industry  Institute  is 
continuing  its  research  on  design  effectiveness.  Phase  I  of  the  research  developed  an 
evaluation  method  and  established  several  design  performance  criteria.  Industrial 
projects  were  investigated  during  phase  II.  In  phase  III,  the  researchers  wish  to 
explore  the  applications  of  the  method  to  commercial  building  and  heavy  civil 
construction.  Included  with  the  general  explanation  of  the  evaluation  method  is  a 
copy  of  the  interview  guide  the  researchers  will  use  to  gather  information  from 
individuals  on  selected  projects. 

Please  do  not  let  the  size  of  this  package  trouble  you.  We  are  only  asking  that 
you  fill  out  a  point  of  contact  form  at  this  time. 

Your  company  is  being  asked  to  volunteer  commercial  building  and/or  heavy 
civil  projects  which  would  provide  useful  information  on  the  design  ev^uation  criteria 
and  sub-criteria  listed  below.  Optimally,  the  projects  volunteered  would  allow  the 
researchers  to  obtain  input  from  the  project's  owner,  designer,  and  constructor.  If 
your  company  has  already  implemented  the  objectives  matrix  on  a  project(s)  a  point  of 
contact  is  requested  so  the  researchers  can  collect  information  conceming  the  strengths 
and  weaknesses  of  the  method.  For  this  portion  of  the  research  all  types  of  projects 
are  welcome. 

Criteria  and  Sub-Criteria  include: 

ACCURACY  OF  THE  DESIGN  DOCUMENTS 

~  Drawing  revisions  /  total  number  of  drawings 

~  Revised  drawings  /  total  number  of  drawings 

~  Specification  revisions  /  total  number  of  specifications 

~  Revised  specifications  /  total  number  of  specifications 

~  Manhours  of  rework  /  total  number  of  manhours 

USABILITY  OF  THE  DESIGN  DOCUMENTS 

—  Drawing  size 

~  Number  of  drawings 

~  Number  of  cross  references 

~  Field  engineering  manhours  /  total  engineering  manhours 

—  Clarity  of  drawings 

~  Completeness  of  drawings 


.'? 


COST  OF  DESIGN  EFFORT 

~  Design  cost  /  construction  cost 

~  Design  manhours  expended  /  design  manhours  budgeted 

CONSTRUCTABILITY 

—  Unrealistic  tolerances  vs.  quality  expectations 

~  Number  of  crafts 

~  Compatibility  with  current  materials  and  technology 

ECONOMY  OF  DESIGN 

~  Building  layout  efficiency 

~  Amount  of  overdesigned  members 

~  Amount  of  overspecified  materials 

PERFORMANCE  AGAINST  SCHEDULE 

~  Percent  of  construction  document  release  dates  attained 

~  Percent  of  intermediate  document  release  dates  attained 

EASE  OF  OCCUPANCY  /  START-UP 

~  Occupancy  preparation  days  required  /  days  budgeted 

~  Number  of  maintenance  personnel  required  /  number  budgeted 

The  graduate  student  conducting  the  interviews  is  Michael  Peek.  If  you  could 
pass  this  package  to  an  appropriate  individual  on  a  suitable  project,  and  return  the 
"point  of  contact"  form,  Mike  will  make  the  next  contact. 

RESEARCH  OFFICE  ADDRESS 

(512)471-4648 

Michael  Peek 

Construction  Industry  Institute 

The  University  of  Texas  at  Austin 

3208  Red  River  Street,  Suite  300 

Austin,  Texas  78705-2650 

Home  Phone  Number 
(512)  345-2640 

Again  we  want  to  emphasize  that  this  package  is  purely  informative,  and 
intended  only  to  give  you  some  insight  into  what  information  the  research  requires. 
Thank  you  for  your  time  and  assistance. 


Sincerely, 
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DESIGN  RESEARCH  PROJECT  III  -  OUTPUTS 

POINT  OF  CONTACT  FORM 


Please  provide  the  following  information  regarding  the  project.  All  data  provided, 
including  company  identification,  will  be  kept  confidential. 


Company  name: 
Point  of  contact: 
Phone  number: 


Project  title  &  description: 


Contact's  address: 


Please  return  the  completed  form  to: 


Michael  Peek 

Construction  Industry  Institute 
The  University  of  Texas  at  Austin 
3208  Red  River  Street,  Suite  300 
Austin,  Texas  78705-2650 


EVALUATION  OF  DESIGN  EFFECTIVENESS 

(DESIGN  RESEARCH  PROJECT  -  HI) 


Purpose  of  the  Study 

Problem  description 

Phases  I  and  n  of  the  "Outputs"  research  effort  have  a )  adapted  the  objectives  matrix 
technique  as  a  method  for  evaluating  design  effectiveness  and  b)  established 
guidelijies  for  utiUzing  the  method.  Unless  this  method  is  implemented  on  actual 
construction  projects,  no  benefits  will  be  realized. 

Problem  scope 

The  purpose  of  this  research  effort  is  two-fold:  (1)  to  encourage  and  assist  Cn 
member  companies  in  implementing  the  objectives  matrix  for  evaluating  design 
effectiveness,  and  (2)  to  apply  the  method  to  additional  areas  of  construction,  such  as 
commercial  building  and  heavy  civil  projects. 

Expected  product 

The  expected  product  of  this  research  effort  will  include  reports  to  the  CU 
membership  on  the  results  of  implementing  the  objectives  matrix  on  actual  projects. 
Included  will  be  an  evaluation  of  the  method's  strengths  and  weaknesses,  and 
recommended  changes  for  implementing  and  using  the  objectives  matrix. 
Additionally,  a  report  will  be  prepared  on  the  application  of  the  objectives  matrix  to 
other  areas  of  construction,  (only  piping  was  studied  in  phases  I  &  II  of  the  "outputs" 
research). 


THE  OBJECTIVES  MATRIX 

Explanation  of  the  Objectives  Matrix 

The  following  is  a  brief  explanation  of  the  Objectives  Matrix. 

Select  the  performance  criteria  '  ■ 

The  first  step  in  constructing  the  matrix  is  to  select  the  criteria  which  pertain  to  the 
performance  goals.  This  requires  the  identification  of  key  indicators  of  performance. 
These  indicators  should  relate  to  such  factors  as  quality,  production  time,  ability  to 
meet  deadlines,  waste,  etc.  Each  of  these  criterion  and  how  it  will  be  measured 
should  be  well  defined. 

Select  performance  scores 

Each  column  in  the  matrix  represents  one  performance  criterion.  Each  row  represents 
a  specific  level  of  performance.  Performance  levels  range  from  a  low  score  of  zero, 
to  a  high  score  of  ten.  A  score  of  three  is  assumed  to  be  the  average  in  order  to 
provide  more  room  for  improvement.  Each  score  corresponds  to  a  value  under  the 
criteria,  and  it  is  these  values  which  form  the  performance  scale  for  the  criteria. 
Benchmark  scores  are  designated  for  performance  levels  zero,  three,  and  ten. 

Level    0  -  The  lowest  level  of  acceptable  performance  over  a  recent  period, 

say  of  two  years,  under  normal  operating  conditions. 
Level    3  -  The  current  level  of  performance. 

Level  10  -  The  desired  level  of  performance.  This  should  represent  a  level  of 
performance  that  can  be  realistically  obtained  in  tiie  forseeable 
future  with  essentially  the  same  resources  available  at  the  present 
time. 

Once  these  benchmark  scores  are  established,  the  increments  between  them  are 
assigned  values. 

Assignment  of  weights 

Weighting  of  the  performance  criteria  provides  a  means  of  directing  management's 
attention  toward  those  criteria  which  have  the  most  impact  on  performance.  All  the 
criteria  selected  do  not  have  an  equal  impact  on  overall  performance.  A  total  of  one 
hundred  points  should  be  distributed  among  the  performance  criteria  to  reflect  the 
contribution  of  each  criterion  to  the  overall  performance.  One  aspect  of  this  study  is 
to  consult  with  industry  personnel  to  determine  what  weights  should  be  assigned  to 
the  chosen  criteria. 

Calculate  the  performance  index 

The  final  phase  ties  together  the  criteria  scores  and  weights  to  determine  the  overall 
performance  index.  The  performance  index  is  calculated  by  multiplying  the  score 
(from  zero  to  ten)  corresponding  to  the  measured  performance  level  for  each  criterion 


by  its  weight,  and  adding  the  products.  The  figure  obtained  is  entered  in  the  box  at 
the  bottom  right  of  the  matrix,  and  represents  the  performance  index  for  a  particular 
evaluation  period. 


Objectives  Matrix  example 

An  example  of  an  objectives  matrix  is  attached  as  an  exhibit. 

Evaluation  criteria  and  sub-criteria 

In  addition  to  adapting  the  objectives  matrix  for  use  in  the  construction  industry, 
phase  I  established  seven  performance  criteria  applicable  to  all  types  of  construction 
projects.  Sub-criteria  were  then  developed  for  each  criterion. 

Accuracy  of  the  Design  Documents  (Accuracy) 

~  Drawing  revisions  /  total  number  of  drawings  (Drawing  Revisions) 

~  Revised  drawings  /  total  number  of  drawings  (Revised  Drawings) 

~  Specification  revisions  /  total  number  of  specifications  (Specification  Revisions) 

~  Revised  specifications  /  total  number  of  specifications  (Revised  Specifications) 

~  Manhours  of  rework  /  total  number  of  manhours  (Rework) 

Usability  of  the  Design  Documents  (Usability) 

—  Drawing  size  (Drawing  Size) 

—  Number  of  drawings  (  Number  of  Drawings) 

~  Number  of  cross  references  between  drawings  and  specifications  (Cross 

References) 
~  Field  engineering  manhours  /  total  engineering  manhours  (Field  Engineering) 
~  Clarity  of  drawings  (Clarity) 
~  Completeness  of  drawings  (Completeness) 

Cost  of  the  Design  Effort  (Cost) 

~  Design  cost  /  construction  cost  (Design  Cost) 

—  Design  manhours  expended  /  design  manhours  budgeted  (Design  Manhours) 

ConstructabiUty  (Constructability) 

~  Unrealistic  tolerances  vs.  quality  expectations  (Tolerances) 

~  Number  of  crafts  (Different  Crafts) 

~  Compatibility  with  current  materials  and  technology  (Materials  and  Technology) 

Economy  of  Design  (Economy) 

—  Building  layout  efficiency  (Layout) 

~  Amount  of  overdesigned  members  (Oversized  Members) 
~  Amount  of  overspecified  materials  (Overspecified  Materials) 

Performance  Against  Schedule  (Schedule) 

~  Percent  of  construction  document  release  dates  attained  (Document  Release) 

~  Percent  of  intermediate  release  dates  attained  (Intermediate  release) 
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Ease  of  Occupancy  /  Start-up  (Occupancy) 

~  Occupancy  preparation  days  required  /  days  budgeted  (Occupancy  Days) 
~  Number  of  maintenance  personnel  required  /  number  budgeted  (Maintenance 
Personnel) 


Sub-Criteria  Definitions 

Accuracy:  Relates  to  the  two  most  common  design  documents,  drawings  and 
specifications. 

Drawings  and  Specifications:  Refers  to  all  drawings  and  specifications  that  are 
necessary  to  construct  the  project. 

Revision:  Is  defined  as  a  change  in  a  drawing  or  specification  that  occurs  after 
construction  of  the  project  has  begun.  Revisions  that  occur  during  the  design  phase 
or  bid  phase  will  not  be  counted. 

Manhours:  Refers  to  all  manhours  required  for  the  construction  phase  of  the  project. 

Rework:  Is  defined  as  a  change  made  to  a  finished  piece,  or  a  portion  a  finished  piece 
of  construction.  It  can  also  be  any  additional  work  performed,  but  not  stipulated  on 
the  design  documents.  Rework  may  be  repeated  on  the  same  item  on  more  than  one 
occasion.  Rework  occurring  only  because  of  design  document  changes,  errors  or 
omissions  will  be  counted  here.  Rework  due  to  an  error  in  a  drawing,  a  change  to  a 
drawing  or  a  late  drawing  will  be  included  here.  Rework  due  to  a  misinterpretation 
of,  or  conflict  with,  the  design  documents  will  be  counted.  Rework  due  to  worker 
error  will  not  be  counted. 

Usability:  Is  best  reflected  by  the  content  and  format  of  the  design  documents  and  the 
engineering  work  required  at  the  construction  site. 

Drawing  Size:  This  element  is  rated  on  a  scale  of  zero  to  ten,  and  should  be  given  a 
high  rating  when  the  physical  size,  scale,  and  clarity  of  the  drawings  were  right  for 
the  work  being  performed.  A  low  score  should  be  given  if  the  drawing  size  and  scale 
makes  them  difficult  to  read. 

Cross  References:  This  item  is  rated  on  a  scale  of  zero  to  ten.  If  the  design 
documents  contain  an  appropriate  amount  of  cross  references,  that  is  if  all  necessary 
referencing  between  drawings,  specifications,  and  drawings  and  specifications  are 
included,  then  a  high  score  should  be  awarded. 

Clarity:  Is  rated  on  a  scale  of  zero  to  ten.  This  item  deals  with  how  well  the  drawing 
explains  what  is  to  be  done.  If  what  is  to  be  done  is  unclear,  then  a  low  score  should 
be  awarded.  A  drawing  that  leaves  no  room  for  misinterpretation  should  be  given  a 
ten.  A  drawing  that  could  easily  mean  several  different  things  should  be  given  a  zero. 
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Completeness:  This  item  measures  on  a  scale  of  zero  to  ten,  how  well  the  drawings 
address  what  is  to  be  done.  If  the  drawing  omits  several  important  details,  it  should 
be  given  a  zero.  A  drawing  that  contains  all  the  information  necessary  to  perform  the 
work  should  be  given  a  rating  of  ten. 

Cost:  Cost  refers  only  to  those  costs  necessary  to  perform  the  construction  phase  of 
the  project.  The  total  cost  of  the  construction  phase  includes  the  cost  of  the  design, 
equipment  rental,  installation,  materials,  and  labor. 

Engineering  Manhours:  Manhours  spent  on  actual  design. 

Constructability:  Is  defined  as  the  optimum  use  of  construction  knowledge  and 
experience  in  the  planning,  engineering,  procurement,  and  field  operations  to  achieve 
the  overall  project  objectives. 

Unrealistic  Tolerances:  This  item  is  rated  on  a  scale  of  zero  to  ten,  and  relates  to  any 
tolerance  that  is  either  unachievable  or  much  too  strict  for  the  purpose.  Tolerances  are 
given  a  low  rating  if  they  are  not  strict  enough  to  meet  the  level  of  quality  expected 
for  the  type  of  work. 

Crafts:  Is  rated  on  a  scale  of  zero  to  ten,  and  refers  to  the  number  of  different  crafts 
required  for  the  construction  of  the  project. 

Current  Materials  and  Technology:  Is  rated  on  a  scale  of  zero  to  ten,  and  includes  the 
use  of  items  or  methods  that  are  out  of  date  or  inefficient. 

Building  Layout  Efficiency:  This  item  is  rated  on  a  scale  of  zero  to  ten,  and  refers  to 
how  well  the  building  layout  is  in  relation  to  its  function.  It  takes  into  account  such 
items  as  placement  of  stairways,  elevators,  mechanical  rooms  etc. 

Overspecified  Materials:  Is  rated  on  a  scale  of  zero  to  ten,  and  includes  those 
materials  whose  properties  are  more  than  adequate  for  their  intended  use. 

Overdesigned  Members:  Is  rated  on  a  scale  of  zero  to  ten,  and  refers  to  the  number  of 
structural  members  that  are  much  larger  than  their  intended  load  requires. 

Design  Document  Release  Deadlines:  These  dates  are  stipulated  in  the  original 
schedule  and  include  dates  on  which  information,  drawings,  and  specifications  are  to 
be  made  available  to  the  constructor. 

Intermediate  Release  Dates:  These  dates  area  deadlines  for  which  drawings  and 
specifications  must  be  available  for  procurement,  permits,  or  other  owner  or  designer 
provided  items  that  pertain  to  the  project. 
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INTERVIEW  GUIDE 

FOR 

DESIGN  EFFECTIVENESS 

DESIGN  RESEARCH  PROJECT  III  -  OUTPUTS 


Richard  L.  Tucker 
Michael  A.  Peek 


Construction  Industry  Institute 

The  University  of  Texas  at  Austin 

3208  Red  River  Street,  Suite  300 

Austin,  Texas     78705-2650 


y'i 


PROJECT  DESCRIPTION  SHEET 


1 .  What  type  of  project  is  this? 

2.  How  long  were  the  durations  of  the  design  and  construction  phases  of  this 
project? 

DESIGN  CONSTRUCTION  


What  type  of  contracts  were  used  for  the  design  and  construction  phases  of  this 

project? 

DESIGN  CONSTRUCTION  


In  what  locations  were  the  construction  and  design  phases  of  this  project 

performed? 

DESIGN  CONSTRUCTION  


5 .    What  company  did  the  design  phase  of  this  project? 
DESIGNER  


6.    What  company  did  the  construction  phase  of  this  project? 
CONSTRUCTOR  


7.  What  company  was  the  owner  of  the  finished  product?    How  would  you  rate  the 
communications  b>etween  the  owner,  the  designer,  and  the  construction 
companies? 

OWNER  

Communications  OK  Problem 

8.  Were  union  or  non-union  workers  involved  in  the  construction  of  the  project? 

9.  Was  there  any  overlap  between  the  design  and  construction  of  the  project?  If  so, 
how  long? 

Overlap    Yes  No 

Duration  (months) 

10.  On  a  scale  of  one  to  ten,  how  would  you  arbitrarily  rate  the  drawings  and 
specifications  used  on  the  project? 

1 1 .  What  was  the  total  cost  of  this  project? 
$  

12.  What  other  criteria  or  sub-criteria  would  you  add  to  the  hst  of  criteria  and  sub- 
criteria  discussed  in  the  explanation  of  the  objectives  matrix,  (as  a  measurement  of 
the  design  outputs)?  What  weight  would  you  assign  these  criteria? 


13.  What  existing  information  does  your  company  track  that  could  be  incorporated 
into  a  matrix  of  this  type? 
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ACCURACY  OF  THE  DESIGN  DOCUMENTS 


1 .  Total  number  of  drawings  on  this  project. 
drawings 

2.  Total  number  of  revisions  to  the  drawings  on  this  project. 
drawing  revisions 

3 .  Total  number  of  revised  drawings  on  this  project. 
revised  drawings 

4.  Total  number  of  specifications  on  this  project. 
specifications 

5.  Total  number  of  revisions  to  the  specifications  on  this  project 
specification  revisions 

6.  Total  number  of  revised  specifications  on  this  project. 
revised  specifications 

7 .  Total  number  of  construction  manhours  on  this  project. 
manhours 

8.  Total  number  of  construction  manhours  due  to  rework  on  this  project. 
manhours 


USABILITY  OF  DESIGN  DOCUMENTS 


1 .  Rating  for  drawing  size  for  this  project  (use  a  scale  of  zero  to  ten,  three  is 
average). 

2.  Rating  of  the  number  of  drawings  on  this  project  necessary  for  crew  use  (use  a 
scale  of  zero  to  ten,  three  is  average). 

3.  Rating  of  number  of  cross  references  between  design  documents  on  this  project 
(use  a  scale  of  zero  to  ten,  three  is  average). 

4.  Number  of  field  engineering  manhours  required  for  this  project. 
manhours 

5.  Total  number  of  design  and  field  engineering  manhours  required  for  this  project. 

manhours 

6.  Rating  of  clarity  of  design  documents  for  this  project  (use  a  scale  of  zero  to  ten, 
three  is  average). 

7.  Rating  of  completeness  of  drawings  for  this  project  (use  a  scale  of  zero  to  ten, 
three  is  average). 


COST  OF  DESIGN  EFFORT 


1 .  Total  budgeted  manhours  for  the  design  of  this  project. 
manhours 

2.  Total  expended  manhours  for  the  design  of  this  project. 
manhours 

3.  Total  design  cost  for  this  project. 

$ 

4.  Total  construction  cost  for  this  project. 

$ 


CONSTRUCTABILITY 


1 .  Rating  for  number  of  different  crafts  required  to  complete  this  project  (use  a  scale 
of  zero  to  ten,  three  is  average). 

2.  Rating  for  the  number  of  unrealistic  tolerances  versus  the  quality  expectations  in 
the  design  documents  for  this  project  (use  a  scale  of  zero  to  ten,  three  is  average). 

3.  Rating  for  level  of  compatibility  of  the  design  of  this  project  with  current  materials 
and  technology  (use  a  scale  of  zero  to  ten,  three  is  average). 


1% 

ECONOMY  OF  DESIGN 


1 .  Rating  for  the  amount  of  overdesigned  members  in  this  project  (use  a  scale  of  zero 
to  ten,  three  is  average). 

2.  Rating  for  the  amount  of  overspecified  materials  in  this  project  (use  a  scale  of  zero 
to  ten,  three  is  average). 

3 .  Rating  for  the  level  of  efficiency  in  building  layout  (or  project  layout  for  heavy 
civil  projects)  for  this  project  (use  a  scale  of  zero  to  ten,  three  is  average). 


PERFORMANCE  AGAINST  SCHEDULE 


1 .  Total  number  of  scheduled  release  dates  for  the  design  documents  of  this  project 
scheduled  release  dates 

2.  Total  number  of  the  design  documents  released  on  the  scheduled  release  dates. 
release  dates  met 

3.  Total  number  of  intermediate  scheduled  release  dates  for  the  design  documents  of 
this  project. 

intermediate  release  dates 

4.  Total  number  of  intermediate  scheduled  release  dates  met  as  per  the  scheduled 
dates. 

release  dates  met 
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EASE  OF  OCCUPANCY  /  START-UP 


1 .  Total  number  of  days  required  to  prepare  for  occupancy  /  use  after  construction 
was  completed. 

days 

2.  Total  number  of  days  budgeted  to  prepare  for  occupancy  /  use. 
days 

3 .  Total  number  of  maintenance  personnel  manhours  required  to  prepare  for 
occupancy  /  use  of  the  building  /  structure. 

manhours 

4.  Total  number  of  maintenance  personnel  manhours  budgeted  to  prepare  for 
occupancy  /  use  of  the  building  /  structure. 

manhours 


WEIGHT  ESTIMATES  FOR  CRITERIA  AND  SUB-CRITERIA 


EVALUATION  OF  DESIGN  CRITERIA 
RANK 

Accuracy  of  the  design  documents 

Usability  of  the  design  documents 

Cost  of  design  effort 

Constructability 

Economy  of  design 

Performance  against  schedule 

Ease  of  occupancy  /  start-up 


WEIGHT 


SUM  =     100 


EVALUATION  OF  DESIGN  SUB-CRITERIA 
RANK 

ACCURACY  OF  THE  DESIGN  DOCUMENTS 

Drawing  revisions  /  total  number  of  drawings 

Revised  drawings  /  total  number  of  drawings 

Specification  revisions  /  total  number  of  specifications 

Specifications  revised  /  total  number  of  specifications 

Manhours  of  rework  /  total  number  of  manhours 


WEIGHT 


SUM  =     100 


USABILITY  OF  THE  DESIGN  DOCUMENTS 
Drawing  size 
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RANK  WEIGHT 

USABILITY  (continued) 

Number  of  drawings  

Number  of  cross  references  

Field  engineering  manhours  /  total  engineering  manhours  

Qarity  

Completeness  of  drawings  

SUM  =     100 


CX)ST  OF  DESIGN  EFFORT 

Design  cost  /  construction  cost 

Design  manhours  expended  /  design  manhours  budgeted 


SUM  =     100 


CONSTRUCTABILITY 

Unrealistic  tolerances  vs.  quality  expectations 

Number  of  crafts 

Compatibility  with  current  materials  and  technology 


SUM  =     100 


ECONOMY  OF  DESIGN 
Building  layout  efficiency 
Amount  of  overdesigned  members 
Amount  of  overspecified  materials 


SUM=    100 


RANK  WEIGHT 


PERFORMANCE  AGAINST  SCHEDULE 

Percent  of  construction  document  release  dates  attained 

Percent  of  intermediate  release  dates  attained 


SUM  =     100 


EASE  OF  OCCUPANCY  /  START-UP 
Occupancy  preparation  days  required  /  days  budgeted 
Number  of  maintenance  personnel  required  /  number  budgeted 


SUM  =     100 
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ESTIMATES  OF  INDUSTRY  AVERAGES 
FOR  EVALUATION  CRITERIA 

10  -  Optimal        3  -  Current  performance       0  -  Minimum  acceptable 

ACCURACY  OF  THE  DESIGN  DOCUMENTS 
Drawing  revisions  /  total  number  of  drawings 

10.  --  3."  0.-- 

Revised  drawings  /  total  number  of  drawings 

10."  3."  0.-- 

Specification  revisions  /  total  number  of  sf)ecifications 

10.--  3.-  0.- 

Revised  specifications  /  total  number  of  specifications 

10.-  3.-  0.- 

Manhours  for  rework  /  total  number  of  construction  manhours 

10.-  3.-  0.- 

USABILITY  OF  THE  DESIGN  DOCUMENTS 
Field  engineering  manhours  /  total  engineering  manhours 

10.-  3.-  0.- 

COST  OF  DESIGN  EFFORT 
Design  cost  /  construction  cost 

10.-  3.-  0.- 

Design  manhours  expended  /  design  manhours  budgeted 

10.-  3.-  0.- 
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PERFORMANCE  AGAINST  SCHEDULE 

Percent  of  construction  document  release  dates  attained 

10.--  3.-  0.- 

Percent  of  intermediate  release  dates  attained 

10.-  3.-  0.- 

EASE  OF  OCCUPANCY  /  START-UP 
Occupancy  /  start-up  preparation  time  required  /  preparation  time  budgeted 

10.-  3.-  0.- 

Number  of  maintenance  personnel  required  /  number  budgeted 

10.-  3.-  0.- 
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